Methods and apparatus for detecting and handling split brain issues in a link aggregation group

ABSTRACT

Methods and apparatus for operating a virtual node in a LAG that includes a first virtual node and a second virtual node are disclosed. The first virtual node includes at least a first fellow node and a second fellow node. In one exemplary method, the first fellow node receives, from the second virtual node, first control information comprising a system ID and first configuration information associated with the LAG. The first control information is compared with reference configuration information representing previously established expected configuration information associated with the LAG. Based on that comparison, and at least one additional criterion, a split brain condition may be determined to exist in the LAG. In a complementary fashion, a fellow node of the second virtual node may be configured to alter its transmitted configuration information depending on whether it is able to communicate with its fellow node in the second virtual node.

RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 14/400,619, which was filed on Mar. 4, 2015, which is anational stage application of PCT/IB2012/054444, which was filed Aug.29, 2012, and claims benefit of U.S. Provisional Application 61/647,211,which was filed May 15, 2012, the disclosures of each of which areincorporated herein by reference in their entirety.

TECHNICAL FIELD

The present invention generally relates to link aggregation, and moreparticularly relates to methods and apparatus for detecting and handlingsplit brain conditions in a Link Aggregation Group.

BACKGROUND

Link Aggregation is widely used to aggregate multiple links between apair of nodes, in order to be able to transmit user data on each of thelinks participating a Link Aggregation Group (LAG) (see, e.g., IEEE802.1AX). Aggregating multiple network connections in this fashion canincrease throughput beyond what a single connection can sustain, and/orcan be used to provide redundancy in case of a failure of one of thelinks. The “Distributed Resilient Network Interconnect” (DRNI) (seeClause 8 of IEEE 802.1AX-REV/D0.2) specifies extensions to LinkAggregation in order to be able to use link aggregation on a networkinterface even between more than two nodes, for example between fournodes A, B, C and D as illustrated in FIG. 1. In FIG. 1 and many of thesubsequent figures of this application, the label “DRNI” is used toindicate a LAG that includes the four nodes A, B, C and D.

As shown in FIG. 1, a LAG is formed between Network 1 and Network 2.More specifically, a LAG is formed between LAG virtual nodes 30, 32.First LAG virtual node 30 includes a first node (A) and a second node(B). Second LAG virtual node 32 includes a third node (C) and a fourthnode (D). LAG Nodes A and C are connected as peer nodes, and LAG Nodes Band D are also connected as peer nodes. Within virtual node 30, nodes Aand B are connected as “fellow nodes,” and similarly within virtual node32 nodes C and D are connected as “fellow nodes.” As used in thisapplication, a “LAG virtual node” refers to a DRNI portal in the IEEEdocumentation discussed above (i.e., two nodes that appear as a singlenode to their respective peers). Additionally, the statement thatvirtual node 30 “includes” two nodes A, B means that the virtual node 30is emulated by the nodes A, B. Similarly, the statement that virtualnode 32 “includes” two nodes C, D means that the virtual node 32 isemulated by the nodes C, D.

Multiple nodes participating in the LAG appear to be the same virtualnode with a single System ID to their peering partner in the LAG. TheSystem ID is used to identify each node (e.g., node A, node B, node C,node D). The System ID is included in Link Aggregation Control ProtocolData Units (LACPDUs) sent between the individual nodes of the LAG. It ispractical to use the System ID of one of the fellow nodes as a commonSystem ID for their corresponding LAG virtual node. Thus, as shown inFIG. 1, node A and node B belong to the same Network 1 and they are partof the same DRNI Portal (i.e., the same LAG virtual node 30), and use acommon System ID of “A” for the emulated LAG virtual node 30. Similarly,Nodes C and D of Network 2 are seen as a single LAG virtual node 32 witha System ID “C” by Nodes A and B.

FIG. 1 also shows the DRNI hand-off of a particular service (see bold“service” line in FIG. 1). The service handed-off on an interface may bea Virtual Local Area Network (VLAN), and an identifier for the servicemay be a VLAN Identifier (VID), such as a Service VID (i.e., “S-VID”)(typically identifying services on Network to Network Interfaces (NNIs))or a Customer VID (i.e. “C-VID”) (typically identifying services on Userto Network Interfaces (UNIs)). In the example of FIG. 1, the service ishanded off on the upper link (between nodes A, C), as both Networks 1and 2 have selected the upper nodes as “active gateway nodes” and haveselected the upper link for the service hand-off. Throughout thisapplication, active gateway nodes are shown as having a bold boundary.This gateway functionality is introduced by DRNI for loop prevention.Thus, the nodes B and D block the service from being handed-off betweenthe DRNI and their own respective networks.

There are different types of failures that have to be handled by theDRNI. One of them is a “portal node failure” illustrated in FIG. 2(“portal node” and “LAG node” are being used synonymously in thiscontext). As shown in FIG. 2, Node A experiences a failure and can nolonger communicate with Node B or Node C. In the prior art, Node B wouldstart to use its own System ID for the LAG instead of the formerly usedcommon System ID, which in the Example of FIG. 1 was the System ID ofNode A. Node C is aware of the unreachability of node A, and node D maybe aware of it too. Node C and Node D have to accept the new partnerSystem ID (B) in order to provide LAG connectivity. Graceful name changefrom the old System ID (A) to the new System ID (B) can be applied forsmoother transition and for not dropping and re-establishing an activeaggregation (see, e.g., N. Finn, Graceful Name Change in LACP, Std.contrib. 2011,http://www.ieee802.org/1/files/public/docs2011/axbq-nfinn-graceful-name-change-0511-v1.pdf).

The behavior illustrated in FIG. 2 is problematic though, because it isbased on changing the System ID of the portal based on the System IDs ofthe individual nodes that comprise the portal. Correspondingly, thisprior art solution provides visibility to the individual systems (i.e.,Nodes C and D learn that node A has failed), which goes against a maindesign principle of the DRNI, which is to hide the details of itsinternal systems that provide an external view of a single LAG virtualnode to its peers. Thus, under DRNI principles, it is desirable to avoidSystem ID change even if a node failure occurs.

FIG. 3 shows another failure event, in the case when connectivitybetween nodes on the same portal (i.e., the “portal link”) is broken,causing the link between fellow Nodes A and B to fail. In the prior art,nodes cannot distinguish between portal link and portal node failures,and Node B's reaction to the portal link failure is the same as to theportal node failure explained above (i.e., Node B starts using its ownSystem ID instead of the common System ID). Nevertheless, Node A is upand running and also uses its own System ID in LACPDUs, which is thesame as the common System ID, as illustrated in FIG. 2. Nodes C and Dthen only maintain the links towards the LAG virtual node 30 that usethe common System ID (i.e., the links to Node A in this example). Thelinks to the other node are deactivated by nodes C and D by deactivatingthe links from LAG as illustrated in FIG. 4 (see dotted line betweenNode B and Node D—this notation will be used throughout this applicationto indicate a deactivated link).

The situation caused by the portal link failure of FIGS. 3-4 is referredto as a “split brain” (SB), because both nodes A and B emulating asingle LAG virtual node 30 are up and running but not connected to eachother. If both of the split brain nodes had become active gateway, thena loop would appear. Nevertheless, the peering partner nodes C and D areable to inform the split brain nodes A and B that they are in a splitbrain situation as shown by FIG. 4. That is, both nodes C and D use anLACPDU to inform their respective peer node that a split brain situationhas occurred at the LAG virtual node 30. Thus, neither of the nodestakes over the active gateway role from the other (e.g., Node B does notbecome the active gateway for the service of FIG. 1). Note that if theconnectivity between nodes C and D works properly, they emulate a singlenode LAG virtual node 32 and both of them are aware of the differentSystem IDs received from their respective peer nodes A and B.

If a split brain situation appears on both sides of the LAG at the sametime, then a “Double Split Brain” (DSB) condition (also known as a “DualSplit Brain”) is said to occur (see FIG. 5). If both sides of the LAGare experiencing split brain, then neither LAG virtual node 30, 32 isable to detect the split brain situation of the other LAG virtual node30, 32, because there is no connection within either portal. Thus thenodes of the same portal cannot notify each other of the fact that theyreceive different System IDs in their respective LACPDUs, which was thebasis of prior art single split brain handling. Therefore, all the nodeswill consider their fellow node within the portal to be down, and allthe nodes become active gateway for all services. This results inforwarding loop of data frames as illustrated in FIG. 5. No method isavailable for handling Double split brain situations.

SUMMARY

A number of methods for operating a virtual node in a Link AggregationGroup (LAG) comprising a first virtual node and a second virtual nodeare disclosed. According to a first exemplary method, a method ofoperating a virtual node in a LAG including a first virtual node and asecond virtual node is disclosed. The first virtual node includes atleast a first fellow node and a second fellow node. According to themethod, the first fellow node determines whether the first fellow nodeis in communication with the second fellow node. The first fellow nodealso transmits control information to the second virtual node based onthe determining such that: the control information comprises a system IDand first configuration information associated with the LAG when thedetermining indicates that the first fellow node is in communicationwith the second fellow node; and the control information comprises thesystem ID and second configuration information associated with the LAGwhen the determining indicates that the first fellow node is not incommunication with the second fellow node, the second configurationinformation being different than the first configuration information.

In one example, the first configuration information and the secondconfiguration information are one of configuration keys and servicedigests. In the same or another example, the first configurationinformation and the second configuration information are carried in LinkAggregation Control Protocol Data Unit (LACPDU) messages. Acorresponding network node configured to implement the exemplary methodis also disclosed.

According to another exemplary embodiment, a method of operating avirtual node in a LAG including a first virtual node and a secondvirtual node is disclosed. The first virtual node includes at least afirst fellow node and a second fellow node. According to the method, thefirst fellow node receives, from the second virtual node, first controlinformation comprising a system ID and first configuration informationassociated with the LAG. The first configuration information is comparedagainst reference configuration information associated with the LAG. Ifthe comparison indicates that the first configuration information doesnot match the reference configuration information, a split braincondition is determined to exist in the LAG in response to at least oneof: detecting that the first fellow node is not in communication withthe second fellow node; and the second fellow node being incommunication with the second virtual node and the first configurationinformation not matching second configuration information provided tothe second fellow node by the second virtual node. The referenceconfiguration represents previously established expected configurationinformation associated with the LAG.

In one example, the determining that the split brain condition existsincludes determining that a dual split brain condition exists in the LAGin response to detecting that the first fellow node is not incommunication with the second fellow node. In this example, anon-primary or non-default connecting link that connects the first andsecond virtual nodes may be excluded from the LAG if a dual split braincondition is determined to exist in the LAG.

In one example, the determining that the split brain condition existsincludes determining that a single split brain condition exists in theLAG in response to the second fellow node being in communication withthe second virtual node and the first configuration information notmatching second configuration information provided to the second fellownode by the second virtual node. The first, second and referenceconfiguration information may be one of a configuration key and aconfiguration digest. Also, the configuration information may be carriedin a Link Aggregation Control Protocol Data Unit (LACPDU) message. Acorresponding network node configured to implement this additionalexemplary method is also disclosed.

Of course, the present invention is not limited to the above featuresand advantages. Indeed, those skilled in the art will recognizeadditional features and advantages upon reading the following detaileddescription, and upon viewing the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a LAG supporting a service.

FIG. 2 is a block diagram of a portal node failure in a LAG.

FIG. 3 is a block diagram of a portal link failure in a LAG, causing asplit brain condition.

FIG. 4 is a block diagram of a prior art solution for handling the splitbrain condition of FIG. 3.

FIG. 5 is a block diagram of a double split brain condition.

FIGS. 6-7 are block diagrams of a normal LAG operating state using anovel System ID naming arrangement.

FIG. 8 is a block diagram of a node failure in the LAG of FIGS. 6-7.

FIG. 9 is a block diagram of a normal operating condition for LAG nodesusing a common key.

FIG. 10 is a block diagram of a non-primary LAG node failure in the LAGof FIG. 9.

FIG. 11 is a block diagram of a primary LAG node failure in the LAG ofFIG. 9.

FIGS. 12-13 are block diagrams of single split brain handling in the LAGof FIG. 9

FIG. 14 is a block diagram of a normal operating condition for LAG nodesusing service assignment digests.

FIG. 15 is a block diagram of a LAG node failure in the LAG of FIG. 14.

FIGS. 16-17 are block diagrams of single split brain handling in the LAGof FIG. 14.

FIG. 18 illustrates an exemplary method of operating a virtual node in aLAG to detect split brain conditions.

FIG. 19 illustrates another exemplary method of operating a virtual nodein a LAG to detect split brain conditions.

FIGS. 20-21 are block diagrams of double split brain handling in a LAG.

FIG. 22 is a block diagram of double split brain handling in a fullymeshed DRNI.

FIG. 23 is a block diagram of broken LAG connectivity due to doublesplit brain mishandling.

FIG. 24 illustrates an exemplary method of operating a virtual node in aLAG.

FIG. 25 illustrates another exemplary method of operating a virtual nodein a LAG.

FIG. 26 illustrates an exemplary LAG node.

DETAILED DESCRIPTION

FIGS. 6-7 illustrate a System ID naming method, where a samenode-identifying System ID is used at each LAG virtual node 30, 32. Asshown in FIGS. 6-7, Nodes A and B claim to be the single virtual Node A′in LAG virtual node 30 towards Network 2. That is, both Node C and NodeD receive LACPDUs such that their partner's System ID is A′ as shown inFIG. 6, and they perceive themselves as being connected to a single NodeA′. The operation on the other side is the same, as Node C and Node Dclaim to be a single LAG virtual node 32 toward Network 1, with a SystemID of C′. In one example, each virtual node's System ID is same as thatof an ID of one of the portal nodes implementing the virtual node. Thatis, in one example System ID A′=System ID A and System ID C′=System IDC.

According to the principles of DRNI operation, it is desirable to hidethe internal operation of a LAG virtual node from its peering network,even in case of failures. FIG. 8 illustrates a block diagram of a nodefailure in the LAGs of FIGS. 6-7, in which this desired behavior isfollowed. That is, despite the breakdown of Node A, Node B still usesSystem ID A′ towards the peering network and does not change it to itsown. If System ID A′=System ID A, then Node B still uses System ID Atowards Network 2 despite the breakdown of Node A. Therefore, theexisting split brain handling solution cannot be applied if the DRNIoperational principles are aimed to be applied and maintained all thetime, as the LAG nodes A, B, C, D in FIGS. 6-7 do not change the SystemID they advertise for LAG.

One potential step to minimize the chance of the appearance of a splitbrain situation is to apply protected links within the portal for theinterconnection of the fellow portal nodes (i.e., have protected portallinks between A-B, and between C-D). For example, the fellow portalnodes could be connected to each other by multiple physical linksgrouped by LAG. Other methods that may be used in combination with or asan alternative to this method are described below.

Single Split Brain Avoidance Using Key Change

FIGS. 9-13 illustrate an exemplary method of using key change to detectand address single split brain conditions. In these figures (and inFIGS. 7-8), a double circle around a node indicates that the node is a“primary node” within its respective LAG virtual node 30 or 32.

The method of FIGS. 9-13 is proposed for single split brain handling ifSystem IDs are not changed when failures occur, and they are not changedunder any circumstances. The proposed method relies on changing anAggregation Key if a fellow portal node is perceived to be down. TheAggregation Key (or “Aggregator Key” or “Actor Key” or simply “Key”) isused in LACPDUs transmitted between Nodes A and C, and between Nodes Band D. Referring to FIG. 9, in a normal operating condition each ofnodes A and B use a key “K” in their transmissions to their peer nodes.

FIG. 10 illustrates a non-primary portal node failure in the LAG of FIG.9, where node B fails. If the virtual node's System ID is same as theSystem ID of one of the portal nodes emulating the virtual node, thenthe node having the common System ID is the primary node (e.g., Node Ais primary System ID A′=System ID A). The primary node does not changeits Aggregation Key, even if a non-primary node goes down as illustratedin FIG. 10.

However, if the primary node A goes down, then the non-primary node Bdoes change its Aggregation Key as shown in FIG. 11. That is, if Node Agoes down then Node B changes the key in its LACPDUs to “L” instead ofthe formerly used common key “K.” In order to avoid disruption due tothe key change, the old and the new key values may be carried in LACPDUsduring the change and for a while after it. Thus the LAG can bemaintained if there is a match with either the old or the new key.Notably, the same System ID of A′ continues to be used by Node B,despite Node B perceiving Node A as having failed.

The handling of a single split brain condition by key change, accordingto one exemplary embodiment, is shown in FIGS. 12-13. Referring to FIG.12, if the connectivity within the portal of virtual node 30 is broken,then both portal nodes A, B perceive that the fellow portal node isdown. Neither of Node A or Node B changes its System ID though, as bothcontinue using System ID A′ towards their peer nodes C, D. The primarynode, (i.e., Node A in the example), also does not change its key and itkeeps using the common key “K.” In contrast, Node B, which is anon-primary node, changes its key value to key “L.”

The peer nodes (Node C and Node D) receive different key values inLACPDUs, and use this to detect that a single split brain has occurredat LAG virtual node 30. The link on which the expected key K is receivedis kept in the LAG (i.e., the link between nodes A and C). Those linkswhere the received key differs from the expected one are excluded fromLAG as shown in FIG. 13 (i.e. the link between B and D is excluded—asshown by the dotted line between B and D).

Note that the difference between the node failure and the split brainconditions is that in the case of a node failure a single key isreceived by the peer even if the key is changed. However, in the case ofa split brain scenario multiple different keys are received. Therefore,the peer is able to recognize a split brain situation and inform thenodes being in split brain about the fact they are in split brain asillustrated in FIG. 13.

Single Split Brain Avoidance Based on Service Assignment Change

FIGS. 14-17 illustrate an exemplary method of using service assignmentdigests to detect and address single split brain conditions. Notably, inembodiments that incorporate the digests described below, there is no“primary” and “non-primary” node, and thus there are no double circlesaround any nodes. However, these embodiments utilize the notion of a“default link” and a “non-default link” in a similar fashion.

In certain scenarios (e.g., in cases where LAG congruency is required),information of the actual service assignments to the individual linksthat are aggregated in a LAG could be carried in the exchanged LACPDUs,in the form of digests (see, e.g., Ericsson provisional patentapplication Ser. No. 61/605,818, entitled “Method for EnsuringCongruency in Link Aggregation”). In such a configuration, certainservices are allocated to specific LAG peer links, and digests aretransmitted between LAG nodes to indicate these service assignments.Comparison of the exchanged assignments at the two ends of the LAGenables verification of the congruency of the carried services.

In the following description, a split brain avoidance method relies ondetecting change in the service to link assignment (i.e., differencescompared to service assignment information expected to be received in adigest). As described below in greater detail, by taking serviceassignment digests to represent LAG service assignments in LACPDUs,split brain conditions may be detected based on certain digest changes.

FIG. 14 is a block diagram of a normal operating condition for LAG thatuse service assignment digests. Because all links are operational inFIG. 14, a same service assignment is used throughout the LAG, and asame service digest “G” is transmitted between peer nodes in the LAG.That is, on each aggregated link an additional service digest isexchanged carrying the information on, for example, VLAN Identifier(VID) assignments to physical links. As the VIDs are handed-off in acongruent manner, all the digest values are the same and a same servicedigest “G” is exchanged between both LAG virtual nodes 30, 32 as shownin FIG. 14. The assignment is consistent if a VID is assigned to thesame physical link on both sides of the LAG, and the digest covers theassignments of all VIDs. Thus, the same digest is expected to bereceived as the one that is sent.

FIG. 15 illustrates operation in case of a node failure. In this caseall of the services that were carried by peer link B-D are now beingcarried by peer link A-C. This means a new Digest H will be communicatedto Network 2 indicating the service transfer. The LAG virtual node 32 ofNetwork 2 is also aware of the failure because of losing a link of LAG,therefore, virtual node 32 also makes changes in the appropriatearrangements as to move all the services to link A-C and update thedigest. Thus the digest exchange becomes as depicted in FIG. 15.

An exemplary method of handling a single split brain condition based onchanges in the service assignments is shown in FIGS. 16 and 17. If theconnectivity within the portal is broken (e.g., link between Node A andNode B is broken), both portal nodes A and B perceive that their fellowportal node is down. Each of the aggregated links takes over all theservices that used to be carried on the other link. As a result, Digest“H” sent by Node A is calculated such that all services are carried overthe A-C link. Node B also sends Digest “J” which is calculated such thatall services carried over the B-D link. Thus the portal nodes in thesplit brain send different digests as shown in FIG. 16. Consequently,each of Nodes C and D receive different digests from their peer nodes Aand B over the A-C and B-D links. Nodes C and D can use this informationto determine that their peer is experiencing a split brain condition,and can take appropriate action so that the LAG is split by separatingthe lower ID link (i.e., deactivating the “non-default link” betweenNode B and Node D), and keeping the “default link” in LAG (which is theA-C link between Node A and Node C in the example of FIGS. 16-17). Atthe same time, Nodes C and D inform their respective peer nodes on A andB about the split brain condition at the LAG virtual node 30 (see “SB”notifications in FIG. 17).

With this in mind, FIG. 24 illustrates an exemplary method 300 ofoperating a virtual node in a LAG comprising a first virtual node (e.g.,LAG virtual node 30) and a second virtual node (e.g., LAG virtual node32). As an example, the method 300 will be described with the “firstvirtual node” being LAG virtual node 30, which includes at least a firstfellow node and a second fellow node (e.g., nodes A and B). The firstfellow node determines whether the first fellow node is in communicationwith the second fellow node (step 302), and transmits controlinformation to the second virtual node based on the determining (steps304, 306). The control information includes a system ID and firstconfiguration information associated with the LAG when the determiningindicates that the first fellow node is in communication with the secondfellow node (step 304). The control information comprises the system IDand second configuration information associated with the LAG when thedetermining indicates that the first fellow node is not in communicationwith the second fellow node, with the second configuration informationbeing different than the first configuration information (step 306).

As described above, the first configuration information and the secondconfiguration information may be configuration keys and/or servicedigests. Also, the first configuration information and the secondconfiguration information may optionally but advantageously be carriedin LACPDU messages.

Single and Double Split Brain Handling

FIGS. 18-19 illustrate exemplary detailed methods of operating a virtualnode in a LAG to detect split brain conditions. In each embodiment, eachLAG node (A, B, C, D) is configured to expect certain information fromits LAG peer, and once detected, the double split brain condition may beaddressed by excluding a connecting peer link.

FIG. 18 illustrates an exemplary method 100 that facilitates single ordouble split brain handling, based on the reception of a LACPDU from apeer node. For the sake of example, the method 100 will be describedfrom the standpoint of Node D of LAG virtual node 32.

A check is performed to determine if control information has beenreceived from the other virtual node (e.g. from node 30) (step 102). Theinformation may be received, for example, in a LACPDU. The controlinformation includes a system ID and also first configurationinformation. A check is performed to determine if the firstconfiguration information matches reference configuration information(step 104). The reference configuration represents previouslyestablished expected configuration information associated with the LAG(e.g., an expected key). If the first configuration information does notmatch the reference configuration information, a check is performed todetermine if Node D is able to communicate with its fellow Node C (i.e.,whether node C is reachable) (step 106). If Nodes C and D cancommunicate with each other, then a check is performed to determine ifsecond configuration information is received at fellow Node C (step107). If the Second Configuration Information is not received after adefined period of time, then it may be determined that peer Node A hasfailed (step 108). Otherwise, if the Second Configuration Information isreceived, and does not match the First Configuration Information (e.g.,fellow nodes C and D receive different Configuration Information fromtheir peer nodes A and B) then a single split brain condition isdetermined to have occurred at LAG virtual node 30 (step 109). Based onthis, a check is performed to determine if the link in question is stilla member of the LAG (step 112). In the example of Node D, this would bethe peer link connecting Nodes B and D. If the link is still a member ofthe LAG, then it is excluded from the LAG (step 114), and a LACPDU witha single SB notification is sent to the LAG peer, which would be Node B(step 116).

Referring again to step 106, if Nodes C and D are unable to communicatewith each other, then a double split brain condition is detected (step110), and steps 112-116 are performed in a similar fashion, except thata DSB notification would be provided in step 116 instead of a single SBnotification. Thus, a check is performed to determine if the link inquestion is still a member of the LAG (step 112), which in the exampleof Node D would be the peer link connecting Nodes B and D. If the linkis still a member of the LAG, then it is excluded from the LAG (step114), and a LACPDU with a DSB notification is sent to the LAG peer,which would be Node B (step 116). Note that the method proposed heredoes not modify or spoil the handling of other failure events.

Referring again to step 104, if the expected information is included inthe LACPDU, a check is performed to determine if the fellow node isreachable (step 118). Thus, a check is performed to determine if Node Dcan communicate with Node C. If the nodes C, D can communicate, then aprevious double split brain condition no longer exists. A check istherefore performed to determine if the peer link between Node B andNode D is currently part of the LAG (step 120), and if it is not a partof the LAG it is added to the LAG once again (step 122).

Additionally, the prior art single brain method (in which Node B wouldchange its System ID in the event of a failure), and the method 400 (inwhich a single split brain condition is detected by key change) bothwork fine with the double split brain handling proposed in the method100.

Although the method 100 has been described as being implemented by theNode D (with Node D and Node C both expecting the same referenceconfiguration information from their respective fellow nodes), fordisabling of the non-primary connecting peer link between Nodes B and D(such that the primary peer link between Nodes A and C would bemaintained), it is understood that this is only an example. If themethod 100 was also performed by Node C, for example, Node C would stillreceive its expected information in steps 102-104. This is because NodesA and C would continue using the same information (e.g., a same systemID and/or key), and it is only Nodes C and D that would be usingdifferent information. Therefore, the peer link between Node A and NodeC would not be disabled even in the event of a double split brain. Inone embodiment, all four of the nodes A, B, C and D perform the method100 to provide single and double split brain detection at both LAGvirtual nodes 30, 32.

As discussed above, the reference configuration information couldinclude an aggregation key, for example, to be used for informationverification. The reference configuration information could bedetermined, for example, by an initial configuration action, or it couldbe based on the values carried by LACPDU under normal operation (e.g.,during initial LACPDU exchanges). In the example of FIG. 9, Nodes C andD are both configured to store the expected key K of their peer.Information mismatch of the method described in FIG. 18 would occur ifthe key received in the LACPDU differs from the local expected key.

FIG. 19 illustrates an exemplary method of operating a virtual node in aLAG to detect split brain conditions, in which the ConfigurationInformation is a service assignment digest that indicates the assignmentof services to physical LAG links. The digest may be received in aLACPDU, and the service digest may differ from an expected servicedigest. The double split brain handling mechanism in FIG. 18 relies ondetection of mismatch in the received and the expected service-to-linkassignment.

For the sake of example, the method 200 will also be described from thestandpoint of Node D. Steps 202-212 and 214-222 of FIG. 19 are performedin a similar same fashion as steps 102-112 and 214-222 from FIG. 18,except that the expected information corresponds to a service assignmentdigest (as opposed to a key or System ID). One notable differencebetween the methods 100, 200 is that an additional step 213 is includedin the method 200 for determining whether the LAG peer link in question(e.g., the link B-D connecting Node B and Node D) is the default LAGlink. This step ensure that only the non-default link (i.e., thatbetween Nodes B, D) will be deactivated, and the primary link (i.e.,that between Nodes A, C) will not be deactivated. Thus, if a split brainsituation is anticipated and there is a mismatch in the received and inthe expected digest, then the link has to be excluded from the LAGunless it is the default LAG link. In one example, the default LAG linkis the link having the numerically lowest link ID value, unless anotherlink is configured as the Default LAG link.

While FIGS. 18-19 are illustrated at a greater level of detail, FIG. 25illustrates a generalized method 400 of operating a virtual node in aLAG comprising a first virtual node and a second virtual node. In theexample of the method 400, LAG virtual node 32 is the illustrative“first virtual node” and LAG virtual node 30 will be the illustrative“second virtual node”, although these could reversed in differentsituations. The first fellow node (e.g., node C) receives, from thesecond virtual node 30, first control information comprising a system IDand first configuration information associated with the LAG (step 402).The first configuration information is compared against referenceconfiguration information associated with the LAG (step 404). If thefirst configuration information is different (does not match) thereference configuration information (an answer of “no” to step 404),there is a determination of whether the node (node C) is incommunication with its fellow node (e.g., node D) (step 406). If not,then a double split brain condition is determined to exist in the LAG(step 410). If the fellow nodes C,D are in communication, then a singlesplit brain condition is determined to exist in response to the secondfellow node being in communication with the second virtual node (step407) and the first configuration information not matching secondconfiguration information provided to the second fellow node by thesecond virtual node (step 409). Step 408 represents detection of a nodefailure (e.g., node A). The reference configuration may representpreviously established expected configuration information associatedwith the LAG.

The determining that the split brain condition exists may includedetermining that a dual split brain condition exists in the LAG inresponse to detecting that the first fellow node is not in communicationwith the second fellow node (step 406). In this example, a non-primaryor non-default connecting link that connects the first and secondvirtual nodes from the LAG may be excluded if a dual split braincondition is determined to exist in the LAG.

The determining that the split brain condition exists may includedetermining that a single split brain condition exists in the LAG inresponse to the second fellow node being in communication with thesecond virtual node and the first configuration information not matchingsecond configuration information provided to the second fellow node bythe second virtual node (step 407, 409).

In both of the methods 300, 400, the Configuration Information may beeither a configuration key or a configuration digest, and may be carriedin a Link Aggregation Control Protocol Data Unit (LACPDU) message ifdesired.

FIGS. 20-21 are block diagrams of double split brain handling in a LAG,based on System IDs, using the method 100. Referring to FIG. 20, thelink connecting Node A and Node B has been lost, and Node B starts usinga new System ID. Similarly, the connection between Nodes C and D hasbeen lost, and Node D starts using a new System ID. Nodes B and Dtherefore do not receive their expected information (steps 102, 104).Also, each of Nodes B and D are unable to reach their fellow Nodes A andC, respectively (step 106). Therefore, a double split brain condition isdetected (step 110), and a LACPDU with a double split brain indicationis sent from Node B to Node D, and a similar LACPDU is sent from Node Dto Node B (step 116) (see “DSB” notifications of FIG. 21).

From the standpoint of Nodes A and C, each of these nodes receives thesame System ID from their LAG peer (i.e., from each other), that is thelocally expected basic System ID. Therefore, the peer link between nodesA and C is kept in the LAG group, and the peer link A-C continuouslyprovides connectivity between Network 1 and Network 2. However, theSystem ID received in LACPDUs by nodes B and D does not match with thelocally expected basic System ID, and Nodes B and C have lostconnectivity to their fellow portal node. Therefore, nodes B and Ddeclare a double split brain situation and remove the peer link B-Dbetween them from the LAG group (see steps 112-114 discussed above).Furthermore, Nodes B and D may optionally send a DSB notification toeach other (step 116). Note that the removal of the link B-D from theLAG group is a local decision, and does not require any agreement withany other node.

FIG. 22 illustrates the handling of double split rain by the proposedmethods 100, 200 in another example, where a full mesh topology is usedamong the nodes participating in the LAG. In this example, it is onlythe link between nodes A and C that is kept in the LAG group due to themethods proposed here. All the other links are deactivated by removingthem from the LAG group, thus ensuring loop-free connectivity (seedotted lines between the nodes A, B, C, D).

Note that if the topology is not a full mesh, then the connectivityformerly provided by the LAG can be broken due to double split brain ifthe proposed method is applied without careful configuration. In theexample double split brain scenario illustrated of FIG. 23, the SystemID received in the LACPDUs does not match the locally expected basicSystem ID in either of the LAG virtual nodes, and neither node can reachits fellow portal node. Therefore, both links are removed from the LAGgroup, and no connectivity is provided between Network 1 and Network 2.In order to avoid such broken connectivity, and the loops that accompanyprior art DSB conditions (see FIG. 5), the nodes A, B and also the nodesC, D may be configured to use a common System ID as discussed above thatdoes not change in the event of failures.

FIG. 26 illustrates an exemplary LAG node 800 operative to implement themethods described above. The LAG node 800 includes a processor 802operable to implement one or more of the methods described above. Theprocessor 802 is operatively connected to non-transitory memory 804 andat least one input/output device 806 for communicating with its fellowLAG node (within the same LAG virtual node 30 or 32), and with its peerLAG node (in the remote other LAG virtual node). The memory 804 may beused to store the Configuration Information discussed above, forexample. The processor 802 may be implemented by one or more processingcircuits (e.g., microprocessors, hardware, firmware, or a combinationthereof). The memory 804 may include program instructions for SplitBrain Handling, that when run on a LAG node configures the LAG node toperform one or more of the methods described above.

Exemplary Advantages

Some exemplary single split brain handling methods proposed above canavoid changing the System ID in case of any failure event, and cantherefore meet DRNI requirements by hiding portal internal details froma LAG peer.

Some exemplary double split brain handling methods proposed aboveprevent frame looping that would otherwise be caused by a DSB conditionin a LAG, where at least four nodes participate in the LAG. Furthermore,the proposed methods do not influence the handling of other failurescenarios.

The foregoing description and the accompanying drawings representnon-limiting examples of the methods and apparatus taught herein. Assuch, the present invention is not limited by the foregoing descriptionand accompanying drawings. Instead, the present invention is limitedonly by the following claims and their legal equivalents.

What is claimed is:
 1. A method of operating a virtual node in a LinkAggregation Group (LAG) comprising first and second virtual nodes,wherein the first virtual node comprises a first fellow node and asecond fellow node, the method comprising: the first fellow node:transmitting first control information to the second virtual node, thefirst control information comprising a system ID and first configurationinformation comprising one of configuration keys and service digests;determining that the connectivity to the second fellow node is broken;and transmitting second control information to the second virtual noderesponsive to determining that the connectivity to the second fellownode is broken, the second control information comprising the system IDand second configuration information that is different than the firstconfiguration information.
 2. The method of claim 1 wherein each of thefirst and second configuration information is associated with the LAG.3. The method of claim 1 wherein one or both of the first and secondconfiguration information is carried in Link Aggregation ControlProtocol Data Unit (LACPDU) messages.
 4. A network node configured foruse as a first fellow node in a Link Aggregation Group (LAG) comprisingfirst and second virtual nodes, wherein the first virtual node comprisesthe first fellow node and a second fellow node, the first fellow nodecomprising: processing circuitry configured to: transmit first controlinformation to the second virtual node, the first control informationcomprising a system ID and first configuration information comprisingone of configuration keys and service digests; determine that theconnectivity to the second fellow node is broken; and transmit secondcontrol information to the second virtual node responsive to determiningthat the connectivity to the second fellow node is broken, the secondcontrol information comprising the system ID and second configurationinformation that is different than the first configuration information.5. The network node of claim 4 wherein each of the first and secondconfiguration information is associated with the LAG.
 6. The networknode of claim 4 wherein one or both of the first and secondconfiguration information is carried in Link Aggregation ControlProtocol Data Unit (LACPDU) messages.
 7. A method of operating a virtualnode in a Link Aggregation Group (LAG) comprising a first virtual nodeand a second virtual node, the first virtual node comprising a firstfellow node and a second fellow node, the method implemented at thefirst fellow node and comprising: receiving, from the second virtualnode, control information comprising a system ID and first configurationinformation associated with the LAG; comparing the first configurationinformation against reference configuration information associated withthe LAG; and responsive to determining that the first configurationinformation does not match the reference configuration information,determining that a split brain condition exists in the LAG based on oneor both of: a determination that the connectivity between the firstfellow node and the second fellow node is broken; and a determinationthat the second fellow node is in communication with the second virtualnode, and that the first configuration information received from thesecond virtual node does not match other configuration informationprovided to the second fellow node by the second virtual node.
 8. Themethod of claim 7 wherein determining that the split brain conditionexists comprises determining that a dual split brain condition exists inthe LAG responsive to determining that the connectivity between thefirst fellow node and the second fellow node is broken.
 9. The method ofclaim 8 further comprising excluding a non-primary or non-defaultconnecting link that connects the first and second virtual nodes fromthe LAG responsive to determining that a dual split brain conditionexists in the LAG.
 10. The method of claim 7 wherein determining that asplit brain condition exists comprises determining that a single splitbrain condition exists in the LAG based on the second fellow node beingin communication with the second virtual node, and that the firstconfiguration information received from the second virtual node does notmatch other configuration information provided to the second fellow nodeby the second virtual node.
 11. The method of claim 7 wherein the firstconfiguration information is carried in a Link Aggregation ControlProtocol Data Unit (LACPDU) message.
 12. A network node configured foruse as a first fellow node in a Link Aggregation Group (LAG) comprisingfirst and second virtual nodes, wherein the first virtual node comprisesthe first fellow node and a second fellow node, the first fellow nodecomprising: processing circuitry configured to: receive, from the secondvirtual node, control information comprising a system ID and firstconfiguration information associated with the LAG; compare the firstconfiguration information against reference configuration informationassociated with the LAG; and responsive to determining that the firstconfiguration information does not match the reference configurationinformation, determine that a split brain condition exists in the LAGbased on one or both of: a determination that the connectivity betweenthe first fellow node and the second fellow node is broken; and adetermination that the second fellow node is in communication with thesecond virtual node, and that the first configuration informationreceived from the second virtual node does not match other configurationinformation provided to the second fellow node by the second virtualnode.
 13. The network node of claim 12 wherein to determine that a splitbrain condition exists, the processing circuitry is further configuredto determine that a dual split brain condition exists in the LAGresponsive to determining that the connectivity between the first fellownode and the second fellow node is broken.
 14. The network node of claim13 wherein responsive to determining that a dual split brain conditionexists in the LAG, the processing circuitry is further configured toexclude a non-primary or non-default connecting link that connects thefirst and second virtual nodes from the LAG.
 15. The network node ofclaim 12 wherein to determine that a split brain condition exists, theprocessing circuitry is further configured to determine that a singlesplit brain condition exists in the LAG responsive to detecting that thesecond fellow node is in communication with the second virtual node, andthat the first configuration information does not match otherconfiguration information provided to the second fellow node by thesecond virtual node.
 16. The network node of claim 12 wherein the firstconfiguration information is carried in a Link Aggregation ControlProtocol Data Unit (LACPDU) message.